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Abstract - A general lossless joint source- 
channel coding scheme based on linear codes is 
proposed and then analyzed in this paper. It is 
shown that a linear code with good joint spec- 
trum can be used to establish limit-approaching 
joint source-channel coding schemes for arbitrary 
sources and channels, where the joint spectrum 
of the code is a generalization of the input-output 
weight distribution. 

I. Introduction 

In a traditional communication system, source and 
channel coding are treated independently. This is be- 
cause Shannon in 1948 showed that separate source and 
channel coding incurs no loss of optimality provided that 
the coding length goes to infinity, which is now called the 
separation theorem. While this separation is well moti- 
vated for the point-to-point case, it can entail significant 
performance losses in more general scenarios, for exam- 
ple, the transmission of correlated sources over multiple 
access channels [1]. Moreover, even for the point-to-point 
case, it is recently reported by Zhong et al [2] that the 
joint source-channel coding usually works more efficiently 
(in terms of the error exponent) than does the separate 
coding. Therefore, in many applications, it is expected 
to adopt a joint source-channel coding scheme. 

However, for arbitrary sources and channels, how to 
construct a limit-approaching joint source-channel coding 
scheme? To answer this question, we propose and ana- 
lyze a general lossless joint source-channel coding scheme 
based on linear codes in this paper. 

II. Linear Codes, Types and Spectrums 
Before presenting our coding scheme, we first need to 
introduce some new concepts, definitions and notations 
related to linear codes. Let X and y be two finite addi- 
tive groups, then a linear code can be defined by a homo- 
morphism / : X n — » y m , i.e., a map satisfying 

f(x 1 +x 2 )=f(x 1 ) + f(x 2 ), Vx 1 ,x 2 eX n 

where X n and y m denote the direct product of n groups 
X and m groups y, respectively. Note that any permuta- 
tion o~ n on n letters can be regarded as an automorphism 
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on X n , and we denote by £„ an independent uniform 
random permutation on n letters. 

Next, we introduce the concept of types in the meth- 
ods of types [3]. The type of a sequence x in X n is the 
empirical distribution P x on X defined by 

, \ a N(a\x) 
\x\ 

where N(a\x) denotes the number of occurrences of a in 
x and | a; | denotes the length of x. For any distribution 
P on X, the set of sequences of type P in X n is denoted 
by Tp(X). A distribution P on X is called a type of 
sequences in X n if Tj}{X) / 0. We denote by V{X) the 
set of all distributions on X, and denote by V n {X) the 
set of all possible types of sequences in X n . 

Now we start to define the spectrum. The spectrum 
of a set A C X n is the empirical distribution Sx{A) on 
V{X) defined by 

S X (A)(P)^ 1{XGA ^ = P} [ VP 6 *>(*). 

Similarly, the joint spectrum of a set B C X n x y m is the 
empirical distribution S X y{B) on V(X) x P(y) defined 

by 



\B\ 

for all P £ V(X),Q £ V(y). Furthermore, the marginal 
spectrums and the conditional spectrums of B can be de- 
fined as the marginal distributions and the conditional 
distributions of Sxy(B), respectively, that is, 



Sx(B)(P) 



J2 Sxy(B)(P,Q), 
}ev(y) 



S y {B)(Q)^ J2 S xy (B)(P,Q), 
PeV(X) 

a S X y(B)(P,Q) 



S y]x (B)(Q\P) 



S x \y{B){P\Q) 



S X {B){P) 
a S X y(B)(P,Q) 



Sy(B)(Q) ■ 

Note that the conditional spectrum Sy\x{B)(Q\P) (or 
S x \y(B){P\Q)) is well defined only for those P (or Q) 
satisfying S X (B){P) ? (or Sy(B)(Q) ? 0). 
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Then naturally, for a given function / : X n — > y m , 
we may define its joint spectrum Sxy(f), forward con- 
ditional spectrum Sytx(f) and image spectrum Sy(f) 
as S xy (r\(f)), Syi x (i\(f)) and Sy(r\(f)), respectively, 
where rl(/) is the relation defined by {(x, f(x))\x G X n }. 
Careful readers must have noticed that the image spec- 
trum Sy(f) and the joint spectrum Sxy(f) are virtually 
the generalized and normalized versions of the spectrum 
(e.g., [4]) and the input-output weight distribution (e.g., 
[5]), respectively. 

From the above definitions, we can easily obtain the 
following properties. 

Proposition 1 

Sx(x n )(P) = s y (y m )(Q)- 



W*" x y m )(P,Q) = S*(*")(P) • S y (y m )(Q) 
for any P € V n {X) } Q G P m (y), where 



n \ a 



m \ a 



n Qe ^(^(«))!' wq; rw™<?(&))!' 

Proposition 2 Tor a given function f : <Y n — > J 7 " 1 , 

SAfy(^m ° / ° 0"n) = Sxy(f)- 

for any permutations o~ n and o~ m 

Proposition 3 For a given random function F : X n - 
y m , we have 

Pr{F(x) = y} = \y\- m a(F)(P x ,P y ) 

for any x G X n , y G y m , where 



F = E m oFoS, 



(1) 



a(F)(P,Q) 



a E[S X y(F)(P,Q)} 



S X y{X n xy m )(P,Q) 

Proposition 4 If both X and y are Galois field GF(q), 
then there exists a random linear code F : X n — > y m such 
that 

Pv{F(x)=y} = \y\- m 
for any x G X n (x ^ n ), y G y m , or equivalently 

a(F)(P,Q) = l 

for all P G V„(X) (P # Po») and all Q G P m Q>). 

Proposition 5 For a given random linear code F : 
X n ^y m , we have 

Pr{F( Xl )= yi } = \y\- m , 
Pr{F(x 2 )=y 2 \F(x 1 )=y 1 } 
= \y\- m a(F)(P X2 _ Xl ,P y2 _ yi ), 

for any unequal X\,x 2 G X n and any y\,y 2 G y m , where 



III. Lossless Joint Source-Channel Coding 
Based on Linear Codes 

In this section, we will present a lossless joint source- 
channel coding scheme based on linear codes for arbitrary 
sources and channels. To analyze the performance of the 
scheme, we use the information-spectrum methods [6, pp. 
247-268]. In the methods of information-spectrum, a gen- 
eral source is defined as an infinite sequence 



X = {X n = (Xr,X. 



(n) Y (n) 



) )Jn=l 



of n-dimensional random variables X n where each com- 
ponent random variable X^ (1 < i < n) takes values in 
the alphabet X, and a general channel is defined as an in- 
finite sequence W = {W n }'^' =1 of conditional probability 
distribution W n — W n (-\-) satisfying 



£ W n (y\x) = I, 

y e yn 



Vx G X' n 



for each n = 1, 2, ■ • •. Then by the similar notations and 
definitions in [6, pp. 247-268], a general joint source- 
channel coding problem may be described as follows. 

First, let V be a source alphabet (a finite additive 
group). Denote by X and y an input alphabet and an 
output alphabet of a channel, respectively (X and y can 
be arbitrary sets). Suppose that an arbitrary general 
source V = {V n }^ =l and an arbitrary general chan- 
nel W = {W n }^ =l are given. We define an encoder 
ip n : V™ — > X m and a decoder ^ n : y m — > V™ as arbi- 
trary mappings. Setting X m — tp n (V n ) and denoting by 
Y m the output from the channel W m with X m as the 
input, the lossless joint source-channel coding system is 
virtually a Markov chain 



yn X r 



1pn(Y m ). 



We define the error probability e n by 



F(x) = F(x) + Y r 



(2) 



e„ ^ Pr{^" ji MY m )} 

= PvAv)w m {[^-\v)r\ Vn {v)) 1 

that is, e„ is defined as the average error probability with 
respect to the probability distribution Py™ of the source. 
For simplicity, we call a pair (ip n , ip n ) of an encoder and a 
decoder with the error probability e n an (n, m, e„)-code. 
Then for a general source V and a general channel W, we 
define a source V being (i?, e)-transmissible over channel 
W if there exists an (n, m n , e„)-code satisfying 

71 

limsup < R, limsupe„ < e. 

n — >oo Tfln n — >oc 

Now let us consider our coding scheme based on linear 
codes, which is depicted in Figure 2] According to the 
scheme, the (random) encoder $„ is defined by 



where Y m denotes a uniform random vector on y n 



$ n (v) =q n (v,F n {v)) =q n (v, Xi{F n (X n (v))) + U l ), (3) 
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Figure 1: The proposed lossless joint source-channel coding scheme based on linear codes 
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Figure 2: The channel coding scheme and the lossless source coding scheme based on linear codes 



where F n : V" — > U l is a random linear code (U is a 
finite additive group) and F n is defined by Q and (J2J), 
and the quantization q n is a map from V™ x U l to «Y m . 
For comparison, the channel coding scheme [4] and the 
lossless source coding scheme [7, 8] based on linear codes 
are shown in Figure [3 Note that our scheme is in fact a 
combination of the two schemes except that the quanti- 
zation q n is now modified to be correlated with the source 
output V n . 

Next, let us investigate the (R, e)-transmissible condi- 
tion of our scheme based on the random linear code F n . 
To this end, we need Lemma ^ and Lemma |21 

Lemma 1 For a given random linear code F n : V n —> U l 
and a given quantization q n : V" x IA 1 — > X m , we have 

Pr{$ n (i;i) = a:i} = Px»|v»(a!i|«i), (4) 
Pr{$„0 2 ) = X 2 \$n{Vl) = Xi} 

= f3(F n ,q n )(v 1 ,v 2 ,x 1 ,x 2 )P x ™\ v ™{x 2 \v 2 ) (5) 

< (3'(F n ,q n )(v 2 ,x 2 )P Xmlv 4x 2 \v 2 ) (6) 

for any V\,v 2 £ V™ {v\ ^ v 2 ) and any X\,x 2 G X m , 
where <!>„ is defined by @ 7 and 



P X m\ V n(x\v) = 



(7) 



q n x (v,x) = {u € U l \q n (v,u) = x}, 
f3(F n ,q n )(v 1 ,v 2 ,x 1 ,x 2 ) = 



E 



|g„ 1 ('Ui,a;i)||(7 Il 1 (v2,a;2)r 



/3'(F„,g„)(t;2,a;2) = 
max > 

PEJ>„(V)\{P„»} 

»ie«' M 2 6g„ 1 («2, ^2) 



Q(F„)(P,P M2 -^) 



(8) 



(9) 



(10) 



Lemma n is an easy consequence of Proposition and 
hence its proof is omitted here, but it does play an im- 
portant role for coding schemes based on linear codes. 
Though in most proofs of the theorems in information 
theory it is required to generate a sequence of indepen- 
dent variables, we only need pairwise independence in the 
proof of lossless joint source-channel (or channel) coding. 
Therefore, the art of lossless joint source-channel coding 
is how to generate a sequence of mutually independent 
variables subject to a conditional probability, and Lemma 
^ provides a feasible method based on linear codes for 
generating such sequences. 
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Lemma 2 For a given random linear code F n : V" — » U < Pr{($„(f ), Y m ) S n (v)} 



a given quantization q n : V™ x U l — > A" m , fTie average 
error probability e n of the system (Figure^) based on F n 
and q n with optimal decoders satisfies 



£ Pr{($ n K),neS„K)}, (17) 



( 1 W m (Y m \X m ) 1 1 where Y m denotes the channel output corresponding to 

e„ < Pr< — In ■ < — In ■ — the input $„(u). Since the first term on the right-hand 

[n P ym (F m ) n P V n(V n ) side of fT7| can be written as 

+ ±\n(3'{F ni q n )(V n ,X n ) + 1 \+e-^ (11) t/ « 



n J A n (v) = Pr{(^ n (v),Y m ) ? S n (v)} 

where X m denotes the channel input generated randomly = P^{^n(v) — x}W m (y\x) 

subject to the conditional probability distribution Px m |V" (.&,y)£S„(v) 

defined by andY m denotes the output from the chan- {"■) \ ^ „ , , x T j /m/ i n 

iurm a- t y™ a rut? \ ■ a fi a 1^ P X ™\V"{x\v)W {y\x) 

nel W corresponding to X , and p yPniln) J s defined c — ' 

by m- ' ' («.»)**»(*) 

22 P X m Y m\ V n(x,y\v), 

As a "linear code" version of Lemma 3.8.1 in [6], (x.y)$_s n {v) 
Lemma|2]is the most important result on the performance 

of our coding scheme based on linear codes. Its proof is where (a) follows from (jT2J), it follows that 
presented as follows. 

Proof of Lemma \^ By © , we have defined a ran- / , Pv n 

(v)A n (v) 

dom encoder $„ : V™ — > X m based on the random linear " eV " 

code F„ : V" — » U l and the quantization q n , then it fol- = Py™(t>) ^•»y™|v» (a;, 2/|w) 

lows from Lemma^that vev™ (a>,y)<ZS n (v) 

Pt{$ n (v) = x} = P X fn\ v ~(x\v), (12) = Pv-xm Y m(v,x,y) 

Pr{$ n («') = x'|*„(t>) = z} («,«*)«fS» 

< /? (F n ,q n )(v ,x )P X m.\v{x \v ) (13) 

n / / / /\ i / On the other hand, the second term on the right-hand 

for any v, v G V (v =fv ) and any x, x G X . . . ° 

T j.jr; j j , -om im i side of 11711 can be written as 

In order to define a decoder ip n : y — > V , we set 



1 , IF m (y|a;) B n (v) 

— m — ; — — > 

n 



P Y™(V) = Py {(®n(v'),Y m ) G S n (v')} 



-\n-^-- + -lnf3'(F n ,q n )(v,x)+ 1 \, (14) '""'^ 



n P V n(v) n J = p *{®n(v) =x}W m (y\x) 



S n (v) = {(x,y) g rxr|(f,a:,y) G S n }. (15) 



Suppose that a channel output y G ^ m is received, we Pr { $ n(«0 - x'\$ n (v) - x}l{(x',y) G S„(«')} 

define the decoder by v = ipn(y) if there exits a unique x ex 

v G V" satisfying y) G 5 n («). If there exists ( < } P Xm]Vn {x\v)W m {y\x) 

no such t) or exist more than one such v, we define L ' L ' 



v' : v 



"0n(y) = ^0: an arbitrary element in V™. Then for each 
sample encoder <p n generated by $„, there is a well de- ^ (T„,g„)(v ,a; )Pym| V n(a; \v ) 

fined decoder and we denote by ^ n the whole random x'-.(x' ,y)es n (v') 

ensemble of the decoders with respect to the random en- = Px™y m |y" (fj y\ v ) 

coder $ n . The error probability e n with respect to the v'-.v'^v (x,y)ex m xy m 

pair ($„, VPn) of the random encoder and decoder is then r'it? \l ' '\p ( 'I >\ 

given by ' 

e « = £ iV- (16) 



x' :(x' ,y)dS„(v') 



where e n (v) denotes the error probability of a source out 



'eV" (x', y )es„(v>) 



put v G V™, and it can be bounded above in the following Px™\v n 

(x'\v'), 



way: 



where (a) follows from l|12() and l|13fl . Therefore, it follows 



e n (v) < Pr{ Y m ) £ S n (v)} that 

+ Pr| |J {(<f> n (v'),Y m ) e S n (v')}\ J2 P y< v ) B ^ v ) 
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< E E 0'(F n ,q n )(v',x')P xmlvn (x'\v') 

iv»(«)iv«.|v»(i/i«) 



t>ev» 

=: £ P'(F n ,q n )(v',x')P Ym (y)P x ™\vn(x'\v') 

(u' ,x' ,y)ES„ 

<e-~i ]T Pv«(v')W m (y\x')P xm]Vn (x'\v') 

(ti',x',y)ES„ 

< e - " 7 , (19) 
where (a) follows from the inequality 

^(F^Xc'.x'jFMv) < e -" 7 iv,(t/)w m (y|x') 

implied by iJUJ. Hence from JTHJ), l|T7|. l(T%)l and (EH), it 

follows that 



e„ = Pvn(w)e n (w) 

< ^ P V n{v)A n {v)+ Pvn(v)B n (v) 



vEV n u£V" 

< Pr{(V n , X m , Y m ) £ S n } + e- n ~<. 

This completes the proof. □ 
By Lemma |2 we can immediately obtain the main 
result as follows. 

Theorem 1 Let V — {V n }^ =1 be a source and W — 
{W n }'^L 1 a channel. If for a random linear code F n , a 
quantization q n : V™ x U l — * X m and two sequences {m n }, 
{7„} satisfying 

n 

hm sup < R 

n— >oo rrtn 



7„ > 0, 7„ — > 0, and wy n — > oo, as n — > oo 
it holds that 

f i W m (Y m \X m ) 1 , 1 
hm sup Pr< — In — — ttttt^ — < — In ■ 



n P Y ™(Y m ) ~ n Pvn(V n ) 

+ -\nf3'(F n ,q n )(V n ,X n )+ ln \ <e, (20) 
n J 

where X m denotes the channel input generated randomly 
subject to the conditional probability distribution Pjfmiyn 
defined by then the system I Figure. f7)) based on F n 
and q n is (R, e) transmissible, that is, limsup,,^^ e n < e. 

Theorem ^ is an easy consequence of Lemma El and 
hence the proof is omitted here. If V = U is a Galois 
field, it follows from Proposition 0] that for each n there 
exists at least a good linear code /„ : V" — > satisfying 

max a(f n )(P,Q) < 1. (21) 

PET'»(V)\(P n) 

Oefi<")} 



Then by the definition (|10|l in Lemma ^ we have 

- In (i'(f n , q n ){v, x) <0 
n 

for any q„ : U l — > A" m , any v G V" and any a; 6 ,Y m . 
Furthermore, for any conditional probability distribution 
P X m\yn we may easily construct a quantization g n to sim- 
ulate it by J7J), so the system (Figure based on f n 
and g n can achieve the same performance as that indi- 
cated by the Direct theorem in [6, Theorem 3.8.1]. In 
other words, there exist good linear codes for construct- 
ing limit- approaching joint source- channel coding scheme 
for general sources and channels, and to be a good linear 
code, the joint spectrum of /„ should satisfy (|21|l . The 
condition l|21(l may be too strict in practice, so we define 
an asymptotically good linear code by 

limsup-ln max a(f n )(P,Q) < 0. (22) 

Q£V t («)} 



IV. Conclusions and Discussion 

A limit-approaching lossless joint source-channel cod- 
ing scheme based on linear codes is proposed in this pa- 
per, and conditions of good linear codes or asymptotically 
good linear codes are given. The most surprising conclu- 
sion of this work is that the performance of a lossless 
joint source-channel coding scheme based on linear codes 
is determined by the codes ' joint spectrum instead of the 
image spectrum. 

The result of this paper is only a part of our ongoing 
research of joint source-channel coding, and lots of inter- 
esting results can be obtained by using the methods and 
results in this paper. For example, note that the quanti- 
zation in our scheme is arbitrary, we can easily design a 
variable rate coding scheme for memoryless channels by 
constructing an appropriate quantization q n . Besides, the 
method established in this paper can be extended to the 
case of multiple access channels with correlated sources, 
which includes Slepian-Wolf coding as its special case. 
The details of the whole work is presented in [9] . 
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